kermit.columbia.edu/kermit.columbia.edu.tar/kermit.columbia.edu/newsgroups/misc.20041116-20060924/[e-mail]_Sun Oct 16 12:18:03 2005.msg

Internet Message Format | 2006-09-27 | 8KB

Path: newsmaster.cc.columbia.edu!newsfeed.nyu.edu!newsfeed.news.ucla.edu!newsfeed.stanford.edu!news.tele.dk!news.tele.dk!small.news.tele.dk!news-fra1.dfn.de!newsfeed.hanau.net!newsfeed.vmunix.org!peer-uk.news.demon.net!kibo.news.demon.net!news.demon.co.uk!demon!not-for-mail
From: John Hackett <[e-mail]>
Newsgroups: comp.protocols.kermit.misc
Subject: Re: ftp mput /recursive problem....
Date: Sat, 15 Oct 2005 21:57:03 +0100
Lines: 186
Message-ID: <dirqev$7j$1$[e-mail]>
References: <dip2gm$ci$1$[e-mail]> <[e-mail]>
NNTP-Posting-Host: iconinfo.demon.co.uk
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: news.demon.co.uk 1129409823 243 80.177.168.155 (15 Oct 2005 20:57:03 GMT)
X-Complaints-To: [e-mail]
NNTP-Posting-Date: Sat, 15 Oct 2005 20:57:03 +0000 (UTC)
In-Reply-To: <[e-mail]>
X-Accept-Language: en-us, en
User-Agent: Mozilla Thunderbird 1.0.7 (Macintosh/20050923)
Xref: newsmaster.cc.columbia.edu comp.protocols.kermit.misc:15409

Thanks for your reply.

Frank da Cruz wrote:
> On 2005-10-14, John Hackett <[e-mail]> wrote:
> : I am quite new to kermit so I hope this is not a stupid questions but
> : I've spend hours trying to get this to work....
> :
> : I am trying to set up an automated backup from a Fedora3 server to a
> : 500Gb LaCie drive over ethernet. There are about 30k files to transfer.
> :   I know both the server and the Lacie are OK as I have done the
> : complete  transfer with gFTP and all was OK.
> :
> Which version of Kermit are you using?  The 8.0 series has gone through
> several releases, each of which added some features or fixed bugs in the
> ftp client.  If you try the current development build:
>
>   http://www.columbia.edu/kermit/ckdaily.html
>
> do you get the same result?

I was using 8.0.211.

[Is this useful:
211-Features:
  MDTM
  REST STREAM
  SIZE
211 End
   Server does not support AUTH
   Server does not support LANG
   Server does not support MDTM
   Server does not support MLST
   Server does not support PBSZ
   Server does not support PROT
   Server supports REST
   Server does not support SIZE
   Server does not support TVFS
   Server does not support UTF8 ]


I downloaded and installed the ckdaily.html and got exactly the same
problem.

[211-Features:
  MDTM
  REST STREAM
  SIZE
211 End
   Server does not support AUTH
   Server does not support LANG
   Server supports MDTM
   Server does not support MLST
   Server does not support PBSZ
   Server does not support PROT
   Server supports REST
   Server supports SIZE
   Server does not support TVFS
   Server does not support UTF8
]

It seems to be a very specific issue as when I went through the listfile
and the files transferred the problem occurs in exactly the same place
each time.

>
> : I had some problems initially which I solved as follows:
> :
> : set file stringspace 10000000
> : set file listsize 100000
> : set ftp filenames literal
> : set term idle-timeout 100
> : set term idle-action output \32
> :
> The "set term" commands have nothing to do with ftp connections.

It seeed to fix an earlier prolem with timing out - but it may have been
an 'artifact' of other attempts to solve it!

>
> : With kermit I have a script which connects up; goes to the correct
> : directories on each side and can transfers the files. However, when I go
> : to the drive one of two things happens.
> :
> : (1) If I try "ftp mput /recursive /update * " the files appear to
> : transfer. If I repeat it only the changed ones seem to be done again.
> : But on the LaCie the files are invisible - I can't find them anywhere!
> :
> : (2) So I tried this instead:
> :
> : ftp mput /recursive /update /listfile:\m(filelist) - this tranfers all
> : the files (I used find to generate a list of all the files (exc.
> : directories). Now I thik all there files are there and visible but they
> : ar enot in the correct directories! It looks as if kermit is descending
> : but not ascending before creating a new directory.
> :
> There are some different variables to consider.
>
> First, the large number of files.  Do the same things happen if you set up
> a similar scenario but with a much smaller number of files?

No. I set up a smaller test with files in a tree and that worked fine.
It also worked - as far as I can tell - on a much larger directory (2667
files) but fails on a larger one (8200 files).

>
> Second, the recursion.  It takes two to tango.  How do we know the server
> is obeying all of Kermit's directives, or even supports them?  From your
> description it sounds like it's possible that Kermit's CDUP command is not
> having any effect.  Try it by hand, CD to some directory on this connection,
> and then see if CDUP brings you back up.

Yes CDUP works.

>
> Third, the update.  This part seems to be working but you never know.  Do you
> see the same problems if you omit the /UPDATE switch?

Same problem, same place.

>
> Set up a test cases with a similar structure but a small number of files
> and then, using "set ftp debug on" to follow what occurs between Kermit and
> the server, see if you can home in on the problem.
>
> - Frank

It is hard to know how much detail to give. The files DO transfer. What
happens is that the directory structure gets messed up so the expected
number of files/directories at a particular level is not correct and you
have to see where the files have been written.

LISTFILE:

profile/Application Data/Skype/sarah_hac/user1024.dbb
profile/Application Data/Skype/sarah_hac/contactgroup256.dbb
profile/Application Data/Skype/sarah_hac/call256.dbb
profile/Application Data/Skype/sarah_hac/config.lck
profile/Application Data/Skype/sarah_hac/chat512.dbb
profile/Application Data/Skype/sarah_hac/chatmsg256.dbb
profile/Application Data/Skype/sarah_hac/index2.dat

---ALL OK UP TO HERE

--- This is where KERMIT misses a beat and creates dir "Real" under
"Skype" instead if the directory above...

profile/Application Data/Real/rnadmin/rnsystem.dat
profile/Application Data/Real/RealPlayer/db/Backup/iscomplete
profile/Application Data/Real/RealPlayer/db/TRACKS.DBF
profile/Application Data/Real/RealPlayer/db/PLAYTRAX.CDX
profile/Application Data/Real/RealPlayer/db/TRAKINFO.DBF
profile/Application Data/Real/RealPlayer/db/CD.DBF


FROM DEBUG:

HERE IS THE ERROR:
257 "/mirror/sarah/profile/Application Data/Skype/Real" - Directory
successfully created
257 "/mirror/sarah/profile/Application Data/Skype/Real/rnadmin" -
Directory successfully created
...
257 "/mirror/sarah/profile/Application Data/Skype/Real/RealPlayer" -
Directory successfully created
257 "/mirror/sarah/profile/Application Data/Skype/Real/RealPlayer/db" -
Directory successfully created
257 "/mirror/sarah/profile/Application
...

And it remains a directory adrift:
257 "/mirror/sarah/profile/Application Data/Recent" - Directory
successfully created
257 "/mirror/sarah/profile/Application Data/My Documents" - Directory
successfully created
257 "/mirror/sarah/profile/Application Data/My Documents/My Videos" -
Directory successfully created
257 "/mirror/sarah/profile/Application Data/My Documents/My Pictures" -
Directory successfully created
257 "/mirror/sarah/profile/Application Data/My Documents/My
Pictures/2005-09-04" - Directory successfully created
257 "/mirror/sarah/profile/Application Data/My Documents/My Skype
Pictures" - Directory successfully created
257 "/mirror/sarah/profile/Application Data/My Documents/My Music" -
Directory successfully created

John.